Noise level based exposure time control for sequential subimages

ABSTRACT

A sequence of multiple subimages is captured by an imaging sensor organized in multiple subsets of pixels. Each of the subsets of pixels is assigned to capturing a corresponding one of the subimages. For each of the subsets of pixels, a noise level of an output of the pixels of the subset is measured. Depending on the measured noise level, an exposure time for capturing the corresponding subimage is controlled.

FIELD OF THE INVENTION

The present invention relates to a method of imaging a scene and to acorrespondingly configured device.

BACKGROUND OF THE INVENTION

Various kinds of electronic devices, e.g., smartphones, tabletcomputers, or digital cameras, may be equipped with increasinglysophisticated imaging functionalities. Such imaging functionalities forexample include capturing of high dynamic range (HDR) still images orvideos. Here, the term “HDR” means that a dynamic range concerning anability to resolve features in regions of low luminosity and highluminosity of an imaged scene is increased as compared to conventionallycaptured images. This means that a certain contrast level allowing forresolving image details may be attained both in image regions of lowluminosity and image regions of high luminosity, avoiding contrastlimiting effects of under exposure or over exposure.

A typical way of capturing HDR images is to utilize an imaging sensorfor capturing multiple (typically two or three) full resolution imagesin a row, using different exposure times, and then combine these imagesto the HDR image. However, due to the time offset between capturing thedifferent images, the combined HDR image may suffer from blurring orghosting of moving objects in the imaged scene and/or from overallblurring due to movement of the imaging sensor while capturing thedifferent images. While such undesirable effects may in principle bereduced by decreasing the time offset between capturing the differentimages, such decrease of the time offset may often be limited by theutilized imaging sensor and associated image processing electronics. Forexample, detecting the outputs of all pixels of the imaging sensor andprocessing these outputs to obtain the desired image data may limit thetime offset. In typical devices, the minimum supported time offset maybe in the range 20 to 50 ms.

Accordingly, there is a need for techniques which allow for efficientlycapturing high dynamic range images.

SUMMARY OF THE INVENTION

According to an embodiment of the invention, a method of imaging a sceneis provided. According to the method, a sequence of multiple subimagesis captured by an imaging sensor organized in multiple subsets ofpixels. Each of the subsets of pixels is assigned to capturing acorresponding one of the subimages. For each of the subsets of pixels, anoise level of an output of the pixels of the subset is measured.Depending on the measured noise level, an exposure time for capturingthe corresponding subimage is then controlled.

According to an embodiment, the method further comprises that, for eachof the subsets of pixels, samples of an output signal of the pixelsduring the exposure time are obtained, and the obtained samples arestatistically processed, e.g., by averaging or filtering, to evaluatethe noise level and an averaged output signal. In response to the noiselevel being above a threshold, the exposure time for capturing thesubimage corresponding to this subset may be extended to obtain furthersamples of the output signal of the pixels. In response to the noiselevel being below the threshold, the exposure time for capturing thesubimage corresponding to this subset may be ended. After ending theexposure time of this subimage, the averaged output signal of the pixelsmay be utilized for obtaining image data of the subimage, and thecapturing process may proceed to capturing the next subimage of thesequence by obtaining samples of an output signal of the pixels of thesubset of pixels corresponding to this next subimage. Accordingly, theprocess of obtaining samples for one of the subimages and extending orending the exposure time for capturing this subimage may be iterateduntil all subimages have been captured. In this way, the exposure timeapplied for each of the subimages may be controlled dynamically whilecapturing the subimage.

According to an embodiment, the subimages are combined to an overallimage. The overall image may have a higher pixel resolution than thesubimages. In addition or alternatively, the overall image may have ahigher dynamic range than the subimages.

According to an embodiment, motion in the imaged scene is detected byanalyzing the sequence of subimages. This may for example involveidentifying one or more moving objects in the imaged scene by analyzingand comparing the subimages of the sequence. On the basis of thedetected motion in the imaged scene, blur or ghosting in the overallimage may be compensated. For example, this may involve generating theoverall image in such a way that the outputs of pixels identified asrepresenting the same part of a moving object in different subimages isassigned to the same location in the overall image. For example, in somescenarios a moving object may be at different positions when capturingdifferent subimages of the sequence. To avoid that image datarepresenting the same parts of the moving object appear at differentpositions in the overall image, such image data may be shifted toappropriate positions in the overall image. For example, an averageposition of the moving object may be determined from the positions ofthe moving object as identified from the different subimages, and theimage data corresponding to the moving object may be aligned with thisaverage position.

According to an embodiment, on the basis of one or more motion sensors,such as one or more accelerometers, motion of the imaging sensor may bedetected while capturing the sequence of subimages. On the basis of thedetected motion of the imaging sensor, blur in the overall image may becompensated. For example, a movement of the imaging sensor whencapturing different subimages of the sequence may cause that the samepart of the imaged scene appears at different positions in differentsubimages. By detecting the motion of the imaging sensor, the image datafrom the different subimages may be shifted to appropriate locations inthe overall image to compensate such effects.

According to an embodiment, an imaging area of the imaging sensor isdivided into multiple zones. Each of the subsets of pixels may then beassigned to a corresponding one of the zones. The pixels of each subsetmay then comprise those pixels of the imaging sensor which are locatedin the zone to which the subset is assigned. Further, the pixels of eachsubset may comprise at least one further pixel located in each otherzone of the imaging area. Accordingly, the subsets of pixels may eachpredominantly provide image data from a given zone of the imagingsensor, which means that the exposure time for this zone may be setindividually, while at the same time the subset may also provide someimage data from all other zones, which means that each subimage may atleast be utilized for providing a lower resolution image coveringsubstantially the entire imaging area and a higher resolution imagecovering the given zone of the imaging sensor.

According to a further embodiment of the invention, a device isprovided. The device comprises an imaging sensor and at least oneprocessor. The imaging sensor is organized in multiple subsets ofpixels. The at least one processor is configured to capture a sequenceof multiple subimages by the imaging sensor. Each of the subsets ofpixels is assigned to capturing a corresponding one of the subimages.Further, the at least one processor is configured to, for each of thesubsets of pixels, measure a noise level of an output of the pixels ofthe subset and, depending on the measured noise level, control anexposure time for capturing the corresponding subimage.

According to an embodiment, the at least one processor is configured toperform steps of the method according to the above embodiments.

Accordingly, the at least one processor may be configured to, for eachof the subsets of pixels, obtain samples of an output signal of thepixels during the exposure time, statistically process the obtainedsamples to evaluate the noise level and an averaged output signal, inresponse to the noise level being above a threshold, extend the exposuretime to obtain further samples of the output signal of the pixels, andin response to the noise level being below the threshold, end theexposure time for capturing the subimage.

Further, the at least one processor may be configured to combine thesubimages to an overall image. As mentioned above, the overall image mayhave a higher pixel resolution than the subimages and/or a higherdynamic range than the subimages.

Further, the at least one processor may be configured to, detect motionin an imaged scene by analyzing the sequence of subimages and, on thebasis of the detected motion in the imaged scene, compensate blur in theoverall image.

According to an embodiment, the device further comprises one or moremotion sensors. The at least one processor may then further beconfigured to, on the basis of one or more motion sensors, detect motionof the imaging sensor while capturing the sequence of subimages and, onthe basis of the detected motion of the imaging sensor, compensate blurin the overall image.

According to an embodiment, an imaging area of the imaging sensor isdivided into multiple zones, and the subsets of pixels are each assignedto a corresponding one of the zones. The pixels of each subset may thencomprise those pixels of the imaging sensor which are located in thezone to which the subset is assigned. In addition, the pixels of eachsubset may comprise at least one further pixel in each other zone of theimaging area.

The above and further embodiments of the invention will now be describedin more detail with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates a scenario a device according to anembodiment of the invention.

FIG. 2 schematically illustrates an imaging sensor which may be utilizedaccording to an embodiment of the invention.

FIG. 3 shows a flowchart for illustrating a method according to anembodiment of the invention.

FIG. 4 shows a flowchart for illustrating a process of controllingexposure times which may be applied in method of FIG. 3.

FIG. 5 schematically illustrates a device according to an embodiment ofthe invention.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following, exemplary embodiments of the invention will bedescribed in more detail. It has to be understood that the followingdescription is given only for the purpose of illustrating the principlesof the invention and is not to be taken in a limiting sense. Rather, thescope of the invention is defined only by the appended claims and is notintended to be limited by the exemplary embodiments describedhereinafter.

The illustrated embodiments relate to imaging of a scene, specificallywith the aim of efficiently capturing HDR still images or HDR videos. Inthe illustrated embodiments, this involves utilization of an imagingsensor which is organized in multiple subsets of pixels. The subsets ofpixels are utilized for capturing a sequence of subimages. In thisprocess, each subset captures a corresponding one of the subimages. Forcapturing each subimage, an exposure time is individually controlled.This is accomplished depending on a noise level detected in the outputof the subset of pixels which captures this subimage. The subimages arethen combined to an overall image. In typical scenarios, the overallimage has a higher pixel resolution and a higher dynamic range than thesubimages. In particular, by allowing individually adapted exposuretimes for the different subsets of pixels, the exposure times may beoptimized depending on the level of luminosity in different regions ofthe imaged scene, e.g., corresponding to different zones of an imagingarea of the imaging sensor. Further, the sequence of subimages may beutilized for detecting motion in the imaged scene and compensate blur orghosting effects in the overall image. The overall image may be a stillimage or a video frame of a video. Accordingly, a HDR still image or anHDR video may be obtained in an efficient manner.

FIG. 1 shows an exemplary device 100 which may be utilized for imaging ascene in accordance with the concepts as outlined above. The device 100may for example be a smartphone, a digital camera (e.g., a compactcamera, an action cam, or a life-log camera), a portable media player, atablet computer, a sports computer, a smart watch, or the like. Asillustrated, the device 100 is equipped with a camera 110. The camera110 may support capturing still images and/or videos. For this purpose,the camera 110 provides an imaging sensor (not illustrated in FIG. 1)with a plurality of light sensitive pixels. An output of the pixels isutilized for generating image data, from which still images and/orvideos may be generated. Such processing of the output of the pixels maybe accomplished by corresponding processors of the device 100. However,in some scenarios at least a part of the processing may also beaccomplished externally, e.g., by one or more other devices. Forexample, the device 100 may provide an interface for transferring rawimage data to another device or to a network based service, andprocessing of the raw data, e.g., to combine the subimages to theoverall image, may then be performed by this other device or networkservice.

FIG. 2 schematically illustrates an exemplary imaging sensor 112 of thecamera 110. As illustrated, the imaging sensor 112 includes a pluralityof light-sensitive pixels 114 distributed in an array over an imagingarea of the imaging sensor 112. The imaging sensor 112 may for examplecorrespond to a CCD (Charge Coupled Device) or CMOS (Complementary MetalOxide Semiconductor) image sensor chip. In the illustrated example, thepixels 114 are distributed in a rectangular array. However, it is to beunderstood that other ways of arranging the pixels may be utilized aswell, e.g., based on a hexagonal pattern of pixels. Moreover, it is tobe understood that the illustrated number of the pixels 114 was selectedfor illustration purposes, and that in typical practical implementationsthe imaging sensor may provide a number of pixels which is significantlylarger, e.g., in a range of one million pixels to 50 million pixels oreven more. Further, it is to be understood that each of the pixels maydeliver an output signal representing a corresponding luminosity of oneor more color channels. In the following, it will be assumed that pixels114 of the imaging sensor 112 each provide an output represented bythree color channels, thereby allowing for capturing full color stillimages or videos.

As further illustrated in FIG. 2, the imaging area of the imaging sensor112 is divided into multiple zones (in FIG. 2 separated by dashedlines). When capturing an image, each of the zones will provide imagedata corresponding to a certain part of the imaged scene. Theillustrative example of FIG. 2 assumes six zones arranged in arectangular pattern. However, it is to be understood that in practicalimplementations other numbers of zones and/or other geometricarrangements of the zones may be utilized. Further, rather than definingnon-overlapping zones as illustrated in FIG. 2, it is also possible todefine the zone in such a way that neighboring zones overlap each other.

In accordance with the different zones, the imaging sensor 112 isorganized in multiple subsets of the pixels 114. In the example of FIG.2, the subsets of pixels are indicated by a numeral or letter in thepixels 114. The pixels 114 which are part of a first subset areidentified by numeral “1” or letter “A”. A majority of the pixels 114 ofthe first subset, identified by numeral “1” is located in a first zoneof the imaging area, in FIG. 2 illustrated as being located in an upperleft corner of the imaging area. Further pixels 114 of the first subset,identified by letter “A”, are distributed over all zones of the imagingarea. The pixels 114 which are part of a second subset are identified bynumeral “2” or letter “A”. A majority of the pixels 114 of the secondsubset, identified by numeral “2” is located in a second zone of theimaging area, in FIG. 2 illustrated as being located in an upper middleportion of the imaging area. Further pixels 114 of the second subset,identified by letter “A”, are distributed over all zones of the imagingarea. The pixels 114 which are part of a third subset are identified bynumeral “3” or letter “A”. A majority of the pixels 114 of the thirdsubset, identified by numeral “3” is located in a third zone of theimaging area, in FIG. 2 illustrated as being located in an upper rightcorner of the imaging area. Further pixels 114 of the third subset,identified by letter “A”, are distributed over all zones of the imagingarea. The pixels 114 which are part of a fourth subset are identified bynumeral “4” or letter “A”. A majority of the pixels 114 of the fourthsubset, identified by numeral “4” is located in a fourth zone of theimaging area, in FIG. 2 illustrated as being located in a lower leftcorner of the imaging area. Further pixels 114 of the fourth subset,identified by letter “A”, are distributed over all zones of the imagingarea. The pixels 114 which are part of a fifth subset are identified bynumeral “5” or letter “A”. A majority of the pixels 114 of the fifthsubset, identified by numeral “5” is located in a fifth zone of theimaging area, in FIG. 2 illustrated as being located in an lower middleportion of the imaging area. Further pixels 114 of the fifth subset,identified by letter “A”, are distributed over all zones of the imagingarea. The pixels 114 which are part of a sixth subset are identified bynumeral “6” or letter “A”. A majority of the pixels 114 of the secondsubset, identified by numeral “6” is located in a sixth zone of theimaging area, in FIG. 2 illustrated as being located in a lower rightcorner of the imaging area. Further pixels 114 of the sixth subset,identified by letter “A”, are distributed over all zones of the imagingarea. Accordingly, each of the subsets of pixels has a main focus on acorresponding zone of the imaging area, but there is also an overlapbetween the different subsets with respect to the pixels 114 identifiedby letter “A”, which are distributed over the imaging area. As will befurther explained below, the pixels 114 of each subset which aredistributed over the imaging area allow for providing an image coveringsubstantially the entire imaging area with each subset of pixels.

The subsets of pixels are utilized in a temporal sequence for capturingthe sequence of subimages. In the example of FIG. 2, this may involvecapturing the following temporal sequence of subimages: a first subimagecaptured by only the first subset of pixels, a second subimage capturedby only the second subset of pixels, a third subimage captured by onlythe third subset of pixels, a fourth subimage captured by only thefourth subset of pixels, a fifth subimage captured by only the fifthsubset of pixels, and a sixth subimage captured only be the sixth subsetof pixels.

For each subimage of the sequence, and accordingly for each subset ofpixels, the exposure time is controlled individually on the basis of thenoise level observed on the output of the pixels 114 of this subset. Asa general rule, the noise level depends on the level of luminosity towhich the pixels 114 are subjected. For lower levels of luminosity, thenoise level increases. This may be compensated by increasing theexposure time, i.e., by sampling the output of the pixels over a longertime interval. For example, multiple samples of the output of each pixel114 of the subset may be obtained over the exposure time and thenaveraged or otherwise filtered to obtain an average output with a lowernoise level. The noise level may be evaluated for multiple pixels of thesubset, e.g., all pixels or a representative selection of pixels of thesubset, e.g., by averaging or otherwise statistically processing theobserved variations in the pixel outputs. As a result, a single valuerepresenting the noise level for the subset may be obtained and then beused for setting the exposure time when capturing the subimage.

In some implementations, the noise level may be measured while capturingthe subimage. For example, samples of the pixel output signals may becollected over the exposure time and be utilized to calculate the noiselevel for the subset and an average output for each pixel. As long asthe noise level is above a threshold, the exposure time may be extendedto collect further samples of the pixel outputs. By considering thefurther samples in the calculated noise level and average pixel outputs,a lower noise level and more accurate pixel outputs are obtained. Therecalculated noise level may then again be compared to the threshold.This process may be repeated in an iterative manner until the comparisonyields a noise level below the threshold or a maximum exposure time,e.g., in the range of 50 to 100 ms is reached. The threshold and/ormaximum exposure time may be configurable by the user of the device.Alternatively, the threshold and/or maximum exposure time may bepreconfigured by the manufacturer of the device. The threshold may alsobe context dependent, i.e., depend on one or more parameters associatedwith an operating context of the device. For example, the threshold maydepend on whether it is night or day. Further, the threshold may dependon a location of the device. Further, the threshold may depend on speedof the device, e.g., as measured by one or more sensors. This may forexample allow for distinguishing whether the device is used by a personwho is running or walking or by a person who is standing still. Thethreshold may also depend on a type of the imaged scene, e.g., whethermain elements of the imaged scene correspond to one or more persons orto landscape structures or other elements which are typically static.The threshold may also be derived from user preferences, from a scoringof previously captured image data, from social media data, or the like.By controlling the exposure times for the subsets individually on thebasis of the corresponding observed noise level, optimized exposuretimes for the different zones of the imaging sensor may be obtained. Forexample, if one of the zones is subjected to a high level of luminosityin the imaged scene, the desired noise level may be reached with a shortexposure time. Similarly, if another one of the zones is subjected to alow level of luminosity in the imaged scene, the desired noise level maybe reached with a longer exposure time. Underexposure or overexposure ofthe subimages can thus be avoided.

As mentioned above, the subimages of the sequence are combined to theoverall image. The overall image will typically cover the entire imagingarea of the imaging sensor and have the full pixel resolution offered bythe imaging sensor 112. For generating the overall image, the parts ofthe subimages corresponding to the different zones of the imaging areamay be merged. If there are overlaps of the different zones, anaveraging or other form of blending of the image data in the overlappingareas to minimize artifacts of the merging process. For the same reason,averaging or other blending may also be applied at the border ofnon-overlapping zones. Due to the individually optimized exposure times,the resulting overall will typically have a higher dynamic range thanthe individual subimages and also than an image captured by a singlesimultaneous exposure of all pixels of the imaging sensor 112.

In some scenarios, it may be beneficial to further consider motion inthe imaged scene and/or motion of the imaging sensor when merging thesubimages to the overall image. For example, since the subimages arecaptured at different time instances, there is a risk that a movingobject in the imaged scene causes blur or ghosting in the overall image,e.g., if the positions of the object in different subimages correspondto different positions in the overall image. Further, motion of theimaging sensor while capturing the sequence of subimages may cause arisk that the same elements of the imaged scene appear in differentsubimages but are mapped to different positions in the overall image.

For compensation of the effects of motion in the imaged scene, the timeresolution provided by the sequence of subimages may be utilized. Inparticular, the subimages may be analyzed to identify motion in theimaged scene. This may for example involve identifying one or moremoving objects in the imaged scene. Various kinds of objectidentification algorithms may be applied for this purpose. Further, thedifferent subimages may be compared to each other to identify and/orquantify movements, e.g., by calculating motion vectors. When combiningthe subimages to the overall image, the detected motion in the imagedscene may be considered and compensated. For example, for a movingobject the corresponding image data may be extracted from the differentsubimages and projected to the same position in the overall image. Suchposition may for example correspond to a position obtained by averagingpositions of the moving object as derived from the different subimages.Accordingly, the time offset sequence of subimages may be applied forefficiently avoiding or minimizing effects of blur or ghosting due tomotion in the imaged scene.

For compensation of the effects of motion of the imaging sensor 112itself, outputs of one or more motion sensors (e.g., accelerometers) ofthe device 100 may be utilized. For example, such motion sensors may beutilized to detect and quantify movement (e.g., in terms of direction,distance, and/or speed) of the imaging sensor 112 between time instancesof capturing different subimages and to shift the corresponding imagedata accordingly when combining the subimages to the overall image.Various known image stabilization techniques based on processing timeoffset image sequences may be applied. In addition or alternatively,also the motion sensors may also be utilized for performing mechanicalimage stabilization by actively moving the imaging sensor depending onthe outputs of the motion sensors. Accordingly, by utilizing the outputsof motion sensors, effects of blur or ghosting due to motion in theimaging sensor itself may be efficiently avoided or minimized.

FIG. 3 shows a flowchart which illustrates a method of imaging a scene.The method may for example be implemented in a device equipped with animaging sensor organized in multiple subsets of pixels, such as theabove-mentioned imaging sensor 112. If a processor based implementationof the device is utilized, at least a part of the steps of the methodmay be performed and/or controlled by one or more processors of thedevice.

At step 310, a noise level is measured with respect to each of thesubsets of pixels. The noise level may be measured by statisticallyprocessing outputs of the pixels of the subsets, e.g., by averaging orotherwise filtering a variation of the outputs of multiple pixels of thesubset. Here, both a temporal variation and a variation betweendifferent pixels may be considered.

The noise level may be measure while capturing a subimage by the subsetof pixels. Further, the noise level could also be measured beforecapturing a subimage by the subset of pixels, e.g., in a dedicated noisemeasurement phase.

At step 320, exposure times are controlled depending on the measurednoise levels. In particular, for each of the subsets, a correspondingexposure time is set depending on the corresponding measured noiselevel. With increasing noise level, a higher exposure time may beselected. The exposure times may be controlled in a range between aminimum exposure time, e.g., as minimally supported by the imagingsensor and associated readout and signal processing electronics, and amaximum exposure time, e.g., as configured by the user or devicemanufacturer. In some implementations, the exposure times may becontrolled by an iterative process while measuring the noise level andcapturing a subimage. An example of such iterative process is furtherdescribed in connection with FIG. 4.

At step 330, a sequence of subimages is captured. Each of the subimagesis captured by a corresponding one of the subsets of pixels. At the sametime, the other subsets of pixels may be inactive. Accordingly, only aportion of the imaging sensor may be active when capturing eachsubimage, which means that, as compared to capturing an image by allpixels of the imaging sensor, faster readout of image data and lowermemory requirement for storing the image data may be achieved. In someimplementations, the sequence subimages may be captured in a timeinterval which corresponds to typical exposure times of a still image,e.g., in a time interval ranging from 10 ms to 1 s. The time intervalmay be set depending on the number of subsets of pixels and capturedsubimages and/or depending on configured maximum exposure time for eachsubimage.

An imaging area of the imaging sensor may be divided into multiplezones, and the subsets of pixels may each be assigned to a correspondingone of the zones. The pixels of each subset may then include thosepixels of the imaging sensor which are located in the zone to which thesubset is assigned. Further, the pixels of each subset may also includefurther pixels in the other zones of the imaging area. An example ofsuch organization into subsets of pixels was explained in connectionwith FIG. 2.

At step 340, the subimages of the sequence are combined to an overallimage. The overall image typically has a higher pixel resolution thanthe subimages, e.g., a pixel resolution corresponding to a total numberof pixels of the imaging sensor. Further, the overall image typicallyhas a higher dynamic range than the subimages. This higher dynamic rangemay be achieved by individually optimized exposure times for differentzones of the imaging sensor, which may be subjected to different levelsof luminosity in the imaged scene.

At step 350, blur in the overall image may be compensated. This may beachieved by analyzing the sequence of subimages to detect motion in theimaged scene. On the basis of the detected motion in the imaged scene,blur in the overall image may then be compensated. For example, this mayinvolve mapping image data corresponding to an identified moving objectas extracted from different subimages to a certain position in theoverall image, e.g., to a position determined by averaging positions ofthe moving object as derived from the different subimages.

In some implementations, step 350 may also involve compensating blur ofdue to movement of the imaging sensor itself. This may be accomplishedon the basis of one or more motion sensors, such as accelerometers.These motion sensors may be utilized for detecting motion of the imagingsensor or the device to which the imaging sensor is mounted whilecapturing the sequence of subimages. The detected motion may then beused as a basis for compensating blur in the overall image. Suchcompensation may involve processing of image data of the subimages,e.g., by shifting positions on the overall image to which the image dataare mapped in accordance with the motion of the imaging sensor. Inaddition or alternatively, outputs of the imaging sensors may also beused for physically moving the image sensor while capturing the sequenceof subimages to thereby counteract undesired movement of the imagingsensor, such as shaking or vibrations.

Depending on the underlying application scenario, the overall image maythen be further processed to a still image or a video frame of a video.Capturing a video may involve repeating the steps of the method for eachvideo frame of the video.

FIG. 4 shows a flowchart for illustrating an iterative process ofcontrolling the exposure time while capturing the subimage, which may beapplied within the overall method of FIG. 3.

At step 410, capturing of a given subimage of the sequence, identifiedby index N, starts. The subset of pixels assigned to capturing thissubimage may therefore be selected at this point.

As indicated by step 420, the capturing of the subimage involvesobtaining samples of outputs signals of the pixels of the subset ofpixels.

At step 430, the obtained samples are statistically processed todetermine the noise level and average pixel outputs. The statisticprocessing may for example involve temporal averaging of the outputsignals of the pixels, spatial smoothing of the output signals ofneighboring pixels, and/or temporal and/or spatial averaging ofvariations of the outputs of multiple pixels of the subset.

At step 440, the measured noise level is compared to a threshold, e.g.,as configured by the user or device manufacturer. The threshold may alsobe context dependent. For example, the threshold may depend on whetherit is night or day. Further, the threshold may depend on a location ofthe device. Further, the threshold may depend on speed of the device.The threshold may also depend on a type of the imaged scene. Thethreshold may also be derived from user preferences, from a scoring ofpreviously captured image data, or from social media data.

In response to the noise level not being below the threshold, asindicated by branch “N”, the method returns to step 420 to obtainfurther samples of outputs signals of the pixels of the subset of pixelsand repeat the determination of the noise level and average pixeloutputs taking into account the further obtained samples.

If the comparison of step shows that the noise level is below thethreshold, the method proceeds to step 450, as indicated by branch “Y”.At step 450, the method returns to step 410 to proceed with thecapturing of the next subimage, identified by index N+1, using anotherone of the subsets of pixels.

FIG. 5 shows a block diagram for schematically illustrating a processorbased implementation of a device which may be utilized for implementingthe above-described concepts. For example, the structures as illustratedby FIG. 5 may be utilized to implement the device 100 as illustrated inFIG. 1.

As illustrated, the device 100 includes an imaging sensor, such as theimaging sensor 112. Further, the device 100 may include one or moremotion sensors 120, such as accelerometers. Further, the device 100 mayinclude one or more interfaces 130. For example, if the device 100corresponds to a smartphone or similar portable communication device,the interface(s) 130 may include one or more radio interfaces and/or oneor more wire-based interfaces for providing network connectivity of thedevice 100. Examples of radio technologies for implementing such radiointerface(s) for example include cellular radio technologies, such asGSM (Global System for Mobile Communications), UMTS (Universal MobileTelecommunication System), LTE (Long Term Evolution), or CDMA2000, aWLAN (Wireless Local Area Network) technology according to an IEEE802.11 standard, or a WPAN (Wireless Personal Area Network) technology,such as Bluetooth. Examples of wire-based network technologies forimplementing such wire-based interface(s) for example include Ethernettechnologies and USB (Universal Serial Bus) technologies.

Further, the device 100 is provided with one or more processors 140 anda memory 150. The imaging sensor 112, the motion sensors 120, theinterface(s) 130, and the memory 150 are coupled to the processor(s)140, e.g., using one or more internal bus systems of the device 100.

The memory 150 includes program code modules 160, 170, 180 with programcode to be executed by the processor(s) 140. In the illustrated example,these program code modules include an image capturing module 160, amotion detection module 170, and an image processing module 180.

The image capturing module 160 may implement the above-describedfunctionalities of capturing a sequence of subimages by differentsubsets of pixels of the imaging sensor 112. Further, the imagecapturing module 160 may also implement the above-described control ofexposure times depending on measured noise levels.

The motion detection module 170 may implement the above-describedfunctionalities of detecting motion in the captured scene, e.g., on thebasis of an analysis of the sequence of subimages or on the basis ofoutputs of the motion sensor(s) 120.

The image processing module 180 may implement the above-describedfunctionalities of combining the subimages to an overall image. This mayalso include the compensation of blur or ghosting.

It is to be understood that the structures as illustrated in FIG. 5 aremerely exemplary and that the device 100 may also include other elementswhich have not been illustrated, e.g., structures or program codemodules for implementing known functionalities of a smartphone, digitalcamera, or similar device. Examples of such functionalities includecommunication functionalities, media handling functionalities, or thelike.

As can be seen, the concepts as explained above allow for efficientlycapturing image data. Specifically, by the individual control ofexposure times in connection with the sequential utilization ofdifferent subsets of pixels of the imaging sensor allows for efficientlyproviding still images or videos with low noise, high pixel resolutionand high dynamic range.

It is to be understood that the concepts as explained above aresusceptible to various modifications. For example, the concepts could beapplied in various kinds of devices, in connection with various kinds ofimaging sensor technologies, without limitation to CCD or CMOS pixelarrays, but also including array cameras or stereoscopic cameras.Further, the concepts may be applied with respect to various kinds ofpixel resolution of imaging sensors. Moreover, various ways oforganizing the imaging sensor in multiple subsets of pixels may beutilized.

The invention claimed is:
 1. A method of imaging a scene, the methodcomprising: capturing a sequence of multiple subimages by an imagingsensor organized in multiple subsets of pixels, each of the subsets ofpixels being assigned to capturing a corresponding one of the subimages;and for each of the subsets of pixels: obtaining samples of an outputsignal of the subsets of pixels during the exposure time; processing theobtained samples to evaluate the noise level and an averaged outputsignal; in response to the noise level being above a threshold,extending the exposure time to obtain further samples of the outputsignal of the pixels; and in response to the noise level being below thethreshold, ending the exposure time for capturing the subimage.
 2. Themethod according to claim 1, comprising: combining the subimages to anoverall image.
 3. The method according to claim 2, wherein the overallimage has a higher pixel resolution than the subimages.
 4. The methodaccording to claim 2, wherein the overall image has a higher dynamicrange than the subimages.
 5. The method according to claim 2,comprising: by analyzing the sequence of subimages, detecting motion inthe imaged scene; and on the basis of the detected motion in the imagedscene, compensating blur in the overall image.
 6. The method accordingto claim 2, comprising: on the basis of one or more motion sensors,detecting motion of the imaging sensor while capturing the sequence ofsubimages; and on the basis of the detected motion of the imagingsensor, compensating blur in the overall image.
 7. The method accordingto claim 1, wherein an imaging area of the imaging sensor is dividedinto multiple zones, and wherein the subsets of pixels are each assignedto a corresponding one of the zones.
 8. The method according to claim 7,wherein the pixels of each subset comprise those pixels of the imagingsensor which are located in the zone to which the subset is assigned. 9.The method according to claim 8, wherein the pixels of each subsetcomprise at least one further pixel in each other zone of the imagingarea.
 10. A device, comprising: at least one imaging sensor organized inmultiple subsets of pixels; and at least one processor, the at least oneprocessor being configured to: capture a sequence of multiple subimagesby the imaging sensor, each of the subsets of pixels being assigned tocapturing a corresponding one of the subimages; and for each of thesubsets of pixels: obtain samples of an output signal of the subsets ofpixels during the exposure time; process the obtained samples toevaluate the noise level and an averaged output signal; in response tothe noise level being above a threshold, extend the exposure time toobtain further samples of the output signal of the pixels; and inresponse to the noise level being below the threshold, end the exposuretime for capturing the subimage.
 11. The device according to claim 10,wherein the at least one processor is configured to combine thesubimages to an overall image.
 12. The device according to claim 11,wherein the overall image has a higher pixel resolution than thesubimages.
 13. The device according to claim 11, wherein the overallimage has a higher dynamic range than the subimages.
 14. The deviceaccording to claim 11, wherein the at least one processor is configuredto: by analyzing the sequence of subimages, detect motion in an imagedscene; and on the basis of the detected motion in the imaged scene,compensate blur in the overall image.
 15. The device according to claim11, wherein the device further comprises one or more motion sensors, andwherein the at least one processor is configured to: on the basis of oneor more motion sensors, detect motion of the imaging sensor whilecapturing the sequence of subimages; and on the basis of the detectedmotion of the imaging sensor, compensate blur in the overall image. 16.The device according to claim 10, wherein an imaging area of the imagingsensor is divided into multiple zones, and wherein the subsets of pixelsare each assigned to a corresponding one of the zones.
 17. The deviceaccording to claim 16, wherein the pixels of each subset comprise thosepixels of the imaging sensor which are located in the zone to which thesubset is assigned.
 18. The device according to claim 17, wherein thepixels of each subset comprise at least one further pixel in each otherzone of the imaging area.